Playing with curricular milestones in the educational sandbox: Q-sort results from an internal medicine educational collaborative.

PURPOSE: In competency-based medical education, the focus of assessment is on learner demonstration of predefined outcomes or competencies. One strategy being used in internal medicine (IM) is applying curricular milestones to assessment and reporting milestones to competence determination. The authors report a practical method for identifying sets of curricular milestones for assessment of a landmark, or a point where a resident can be entrusted with increased responsibility. METHOD: Thirteen IM residency programs joined in an educational collaborative to apply curricular milestones to training. The authors developed a game using Q-sort methodology to identify high-priority milestones for the landmark "Ready for indirect supervision in essential ambulatory care" (EsAMB). During May to December 2010, the programs'ambulatory faculty participated in the Q-sort game to prioritize 22 milestones for EsAMB. The authors analyzed the data to identify the top 8 milestones. RESULTS: In total, 149 faculty units (1-4 faculty each) participated. There was strong agreement on the top eight milestones; six had more than 92% agreement across programs, and five had 75% agreement across all faculty units. During the Q-sort game, faculty engaged in dynamic discussion about milestones and expressed interest in applying the game to other milestones and educational settings. CONCLUSIONS: The Q-sort game enabled diverse programs to prioritize curricular milestones with interprogram and interparticipant consistency. A Q-sort exercise is an engaging and playful way to address milestones in medical education and may provide a practical first step toward using milestones in the real-world educational setting.

The development of the emergency medicine milestones.

The Accreditation Council for Graduate Medical Education (ACGME) has outlined its "Next Accreditation System" (NAS) that will focus on resident and residency outcome measurements. Emergency medicine (EM) is one of seven specialties that will implement the NAS beginning July 2013. All other specialties will follow in July 2014. A key component of the NAS is the development of assessable milestones, which are explicit accomplishments or behaviors that occur during the process of residency education. Milestones describe competencies more specifically and identify specialty-specific knowledge, skills, attitudes, and behaviors (KSABs) that can be used as outcome measures within the general competencies. The ACGME and the American Board of Emergency Medicine (ABEM) convened an EM milestone working group to develop the EM milestones. This article describes the development, use within the NAS, and challenges of the EM milestones.

Educational milestone development in the first 7 specialties to enter the next accreditation system.

BACKGROUND: The Accreditation Council for Graduate Medical Education (ACGME) Outcome Project introduced 6 general competencies relevant to medical practice but fell short of its goal to create a robust assessment system that would allow program accreditation based on outcomes. In response, the ACGME, the specialty boards, and other stakeholders collaborated to develop educational milestones, observable steps in residents' professional development that describe progress from entry to graduation and beyond. OBJECTIVES: We summarize the development of the milestones, focusing on 7 specialties, moving to the next accreditation system in July 2013, and offer evidence of their validity. METHODS: Specialty workgroups with broad representation used a 5-level developmental framework and incorporated information from literature reviews, specialty curricula, dialogue with constituents, and pilot testing. RESULTS: The workgroups produced richly diverse sets of milestones that reflect the community's consideration of attributes of competence relevant to practice in the given specialty. Both their development process and the milestones themselves establish a validity argument, when contemporary views of validity for complex performance assessment are used. CONCLUSIONS: Initial evidence for validity emerges from the development processes and the resulting milestones. Further advancing a validity argument will require research on the use of milestone data in resident assessment and program accreditation.

Competency-based medical education: theory to practice.

Although competency-based medical education (CBME) has attracted renewed interest in recent years among educators and policy-makers in the health care professions, there is little agreement on many aspects of this paradigm. We convened a unique partnership - the International CBME Collaborators - to examine conceptual issues and current debates in CBME. We engaged in a multi-stage group process and held a consensus conference with the aim of reviewing the scholarly literature of competency-based medical education, identifying controversies in need of clarification, proposing definitions and concepts that could be useful to educators across many jurisdictions, and exploring future directions for this approach to preparing health professionals. In this paper, we describe the evolution of CBME from the outcomes movement in the 20th century to a renewed approach that, focused on accountability and curricular outcomes and organized around competencies, promotes greater learner-centredness and de-emphasizes time-based curricular design. In this paradigm, competence and related terms are redefined to emphasize their multi-dimensional, dynamic, developmental, and contextual nature. CBME therefore has significant implications for the planning of medical curricula and will have an important impact in reshaping the enterprise of medical education. We elaborate on this emerging CBME approach and its related concepts, and invite medical educators everywhere to enter into further dialogue about the promise and the potential perils of competency-based medical curricula for the 21st century.

Competency-based medical education in postgraduate medical education.

With the introduction of Tomorrow's Doctors in 1993, medical education began the transition from a time- and process-based system to a competency-based training framework. Implementing competency-based training in postgraduate medical education poses many challenges but ultimately requires a demonstration that the learner is truly competent to progress in training or to the next phase of a professional career. Making this transition requires change at virtually all levels of postgraduate training. Key components of this change include the development of valid and reliable assessment tools such as work-based assessment using direct observation, frequent formative feedback, and learner self-directed assessment; active involvement of the learner in the educational process; and intensive faculty development that addresses curricular design and the assessment of competency.

Perspectives on competency-based medical education from the learning sciences.

A central component of competency-based medical education is a framework of higher-order and more fundamental competencies whose purpose is to focus instruction and learning. In the language of the learning sciences, many of these competencies are complex cognitive-perceptual or cognitive-motor skills. Competency-based medical education has been criticized for being reductionistic, that is, for focusing on atomistic skills and failing to capture the essence of professional activities as manifested by complex, integrated capabilities. The value of identifying fundamental skill components is supported by theory and evidence from the learning sciences, however. Complex skills are constructed from fundamental, component skills. Proficient performance of the former is achieved as components are refined and integrated during repeated performance of the skill in a realistic context and as feedback on performance is provided. Competency-based medical education does not propose specific methods for teaching competencies. The learning and instructional sciences, however, posit a number of conditions for learning that support the acquisition of simple skills and their flexible integration into complex capabilities. Learners' motivation and self-regulation skills will also have an impact on the extent to which they engage in learning processes that result in the integration of knowledge and skills into complex competencies.

The role of assessment in competency-based medical education.

Competency-based medical education (CBME), by definition, necessitates a robust and multifaceted assessment system. Assessment and the judgments or evaluations that arise from it are important at the level of the trainee, the program, and the public. When designing an assessment system for CBME, medical education leaders must attend to the context of the multiple settings where clinical training occurs. CBME further requires assessment processes that are more continuous and frequent, criterion-based, developmental, work-based where possible, use assessment methods and tools that meet minimum requirements for quality, use both quantitative and qualitative measures and methods, and involve the wisdom of group process in making judgments about trainee progress. Like all changes in medical education, CBME is a work in progress. Given the importance of assessment and evaluation for CBME, the medical education community will need more collaborative research to address several major challenges in assessment, including "best practices" in the context of systems and institutional culture and how to best to train faculty to be better evaluators. Finally, we must remember that expertise, not competence, is the ultimate goal. CBME does not end with graduation from a training program, but should represent a career that includes ongoing assessment.

Advancing resident assessment in graduate medical education.

BACKGROUND: The Outcome Project requires high-quality assessment approaches to provide reliable and valid judgments of the attainment of competencies deemed important for physician practice. INTERVENTION: The Accreditation Council for Graduate Medical Education (ACGME) convened the Advisory Committee on Educational Outcome Assessment in 2007-2008 to identify high-quality assessment methods. The assessments selected by this body would form a core set that could be used by all programs in a specialty to assess resident performance and enable initial steps toward establishing national specialty databases of program performance. The committee identified a small set of methods for provisional use and further evaluation. It also developed frameworks and processes to support the ongoing evaluation of methods and the longer-term enhancement of assessment in graduate medical education. OUTCOME: The committee constructed a set of standards, a methodology for applying the standards, and grading rules for their review of assessment method quality. It developed a simple report card for displaying grades on each standard and an overall grade for each method reviewed. It also described an assessment system of factors that influence assessment quality. The committee proposed a coordinated, national-level infrastructure to support enhancements to assessment, including method development and assessor training. It recommended the establishment of a new assessment review group to continue its work of evaluating assessment methods. The committee delivered a report summarizing its activities and 5 related recommendations for implementation to the ACGME Board in September 2008.

Using patient care quality measures to assess educational outcomes.

OBJECTIVES: To report the results of a project designed to develop and implement a prototype methodology for identifying candidate patient care quality measures for potential use in assessing the outcomes and effectiveness of graduate medical education in emergency medicine. METHODS: A workgroup composed of experts in emergency medicine residency education and patient care quality measurement was convened. Workgroup members performed a modified Delphi process that included iterative review of potential measures; individual expert rating of the measures on four dimensions, including measures quality of care and educational effectiveness; development of consensus on measures to be retained; external stakeholder rating of measures followed by a final workgroup review; and a post hoc stratification of measures. The workgroup completed a structured exercise to examine the linkage of patient care process and outcome measures to educational effectiveness. RESULTS: The workgroup selected 62 measures for inclusion in its final set, including 43 measures for 21 clinical conditions, eight medication measures, seven measures for procedures, and four measures for department efficiency. Twenty-six measures met the more stringent criteria applied post hoc to further stratify and prioritize measures for development. Nineteen of these measures received high ratings from 75% of the workgroup and external stakeholder raters on importance for care in the ED, measures quality of care, and measures educational effectiveness; the majority of the raters considered these indicators feasible to measure. The workgroup utilized a simple framework for exploring the relationship of residency program educational activities, competencies from the six Accreditation Council for Graduate Medical Education general competency domains, patient care quality measures, and external factors that could intervene to affect care quality. CONCLUSIONS: Numerous patient care quality measures have potential for use in assessing the educational effectiveness and performance of graduate medical education programs in emergency medicine. The measures identified in this report can be used as a starter set for further development, implementation, and study. Implementation of the measures, especially for high-stakes use, will require resolution of significant measurement issues.

The ACGME outcome project: retrospective and prospective.

BACKGROUND: The Accreditation Council for Graduate Medical Education began an initiative in 1998 to improve resident physicians' ability to provide quality patient care and to work effectively in current and evolving healthcare delivery systems. AIMS: This initiative, called the Outcome Project, seeks changes in residency programs that focus education on the competency domains, enhance assessment of resident performance and increase utilization of educational outcomes for improving residents' education. Increased emphasis on educational outcome measures in accreditation is another important goal. RESULTS: A considerable amount of development, dissemination and educational activity has been carried out to support project implementation. Thus far, observed effects include changes to accreditation requirements and information collection and enhancements of the educational environments and curriculum of residency education programs. CONCLUSION: Prospects for meaningful change are good. Further development of assessment methods is needed to advance in-training evaluation of residents and the ACGME goals for utilizing performance data in accreditation and linking education and patient care quality.

Assessing practice-based learning and improvement.

BACKGROUND: Practice-based learning and improvement (PBLI) is 1 of 6 general competencies expected of physicians who graduate from an accredited residency education program in the United States and is an anticipated requirement for those who wish to maintain certification by the member boards of the American Board of Medical Specialties. This article describes methods used to assess PBLI. SUMMARY: Six electronic databases were searched using several search terms pertaining to PBLI. The review indicated that 4 assessment methods have been used to assess some or all steps of PBLI: portfolios, projects, patient record and chart review, and performance ratings. Each method is described, examples of application are provided, and validity, reliability, and feasibility characteristics are discussed. CONCLUSION: Portfolios may be the most useful approach to assess residents' PBLI abilities. Active participation in peer-driven performance improvement initiatives may be a valuable approach to confirm practicing physician involvement in PBLI.

Graduates' perceptions of their clinical competencies in allergy and immunology: results of a survey.

PURPOSE: The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties (ABMS) have identified six areas of general competency. This study surveyed graduates of allergy and immunology training programs about their perceived clinical competency and the adequacy of their subspecialty training. METHOD: In August 2000 and May 2001, a questionnaire was mailed to 373 physicians who had completed a fellowship in allergy and immunology in the United States between 1995 and 2001. Physicians were asked to rate the perceived importance and adequacy of their training in, and their level of competency for, 57 general competencies and subspecialty-specific competencies and procedures. RESULTS: A total of 253 physicians responded (68%). All items in the six ACGME/ABMS general competencies had high ratings (>/= 90%) for perceived importance. One item in the practice-based learning area had low ratings for adequacy of training (57%) and intermediate for competency (75%). Two items in the system-based practice area had low ratings for training (65% and 67%) and intermediate for competency (86% and 88%). Generally, core specialty-specific items (allergic rhinitis, asthma, and urticaria) had high ratings (>/= 90%) for importance, training, and competency. Without exception, items with ratings of less than 70% for adequacy of training also had ratings of less than 90% for competency. CONCLUSION: The general competencies were considered important, but training in system-based practice and practice-based learning may be deficient. Although self-perceived competency in core areas of allergy and immunology was high, weaknesses in training and self-perceived competency in selected areas were identified.

Assessing the ACGME general competencies: general considerations and assessment methods.

The Accreditation Council for Graduate Medical Education's (ACGME's) general competency and outcome assessment initiative (i.e., the ACGME Outcome Project) is an effort to enhance residency education and accreditation effectiveness by increasing emphasis on educational outcomes. The Project is also a response to concerns about new graduates' ability to meet the demands of today's practice environment. The competencies emphasize learning in new domains (e.g., Practice-Based Learning and Improvement and Systems-Based Practice) and more traditional ones (e.g., Patient Care and Medical Knowledge). Outcome assessment will provide evidence of residency program educational effectiveness and information to guide improvement. This paper discusses the development and implementations of assessment methods appropriate to evaluate the performance of residents in each of the core competencies.